This morning (September 3, 2025), someone posted an incident to the Mozilla dev-security-policy list that exposed a serious incident: “Incident Report: Mis-issued Certificates for SAN iPAddress: 1.1.1.1 by Fina RDC 2020.” An obscure CA, Fina RDC 2020, issued certificates containing the IP address 1.1.1.1, used by Cloudflare for encrypted DNS. These certificates should never have been issued.
This IP anchors encrypted DNS (DoH, DoT) and Cloudflare WARP. A mis-issued certificate, combined with a BGP hijack, allows an attacker to intercept traffic before secure tunnels form. Some may argue that an attacker would also need to perform a BGP hijack to exploit such a certificate, but this is no real mitigation. BGP hijacks are a regular occurrence – often state-sponsored — and when combined with a valid certificate, they turn a routing incident into a full man-in-the-middle compromise. Cloudflare documentation:
Mainstream browsers like Chrome and Firefox typically use the domain name cloudflare-dns.com when bootstrapping DoH/DoT, so they would ignore a certificate that only listed IP:1.1.1.1. However, both Google and Cloudflare also support direct IP endpoints (https://8.8.8.8/dns-query and https://1.1.1.1/dns-query). In these cases, an IP SAN certificate would validate just as Alex Radocea’s curl test showed: “subjectAltName: host “8.8.8.8” matched cert’s IP address!”.
The behavior differs slightly between providers:
Google accepts raw-IP DoH connections and does not mandate a specific HTTP Host header.
Cloudflare accepts raw-IP DoH connections but requires the Host: cloudflare-dns.com header.
This means raw-IP endpoints are real-world usage, not just theoretical. If a mis-issued IP SAN cert chains to a CA trusted by the platform (like in Microsoft’s store), exploitation becomes practical.
A Chronic Pattern of Mis-issuance
The incident report cites two mis-issued certs are still valid as of September 3 2025:
This was not an isolated event. A broader search of CT logs reveals a recurring pattern of these mis-issuances from the same CA, indicating a systemic problem.
If these certs surfaced in crt.sh, Cloudflare should have flagged them. Instead, there was no visible remediation.
What’s going wrong?
Are IP SANs being monitored effectively enough to detect a pattern?
Is there a proper pipeline from alerts to action?
Why were these certs still valid after months?
What this means for users
The impact is scoped to users of browsers and applications that rely on the host operating system’s root store, such as Microsoft Edge on Windows. Browsers like Chrome and Firefox, which manage their own root stores. However, because both Google and Cloudflare accept raw-IP DoH endpoints, the risk extends beyond pure edge cases.
For an affected user, at a public hotspot, if a BGP hijack redirects their traffic and the client connects directly to 1.1.1.1, their system checks the cert and sees a valid Cloudflare-like certificate. The attacker succeeds not just in breaking DNS, but in controlling secure web sessions.
This enables:
Silent proxying of all traffic
Token and session theft via impersonation
Decryption of DoH queries and WARP bootstrap (for raw-IP clients)
In-flight alteration of pages and updates
This is the full man-in-the-middle playbook.
A tiny CA with outsized impact
Certificate issuance across the WebPKI is highly skewed, as of today:
Fina RDC 2020 has ~201 unexpired certs, accounting for <0.00002% of issuance. Yet Microsoft trusts it, while Chrome, Firefox, and Safari do not. That asymmetry leaves Edge and Windows users vulnerable. Moreover, Microsoft requires that CAs provide “broad value to Windows customers and the internet community.” https://learn.microsoft.com/en-us/security/trusted-root/program-requirements Fina clearly does not meet that threshold.
Certificate lifespan context
The CA/Browser Forum (April 2025) approved ballot SC-081v3: TLS certificates will be limited to 47 days by March 2029:
The missed opportunity: If short-lived IP certs were required by policy, this incident would have already expired, reducing exposure from months to days. Baseline Requirements currently do not differentiate IP SAN certs from DNS names. If they did, such exposure could be avoided.
On intent and attribution
Attribution in WebPKI failures is notoriously difficult. The noisy, repeated issuance pattern suggests this may have been a systemic accident rather than a targeted attack. Still, intent is irrelevant. A certificate that shouldn’t exist was issued and left valid for months. Once issued, there is no way to tell what a certificate and its key were used for. The governance failure is what matters.
From an attacker’s perspective, this is even more concerning. CT logs can be mined as a reconnaissance tool to identify the weakest CAs – those with a track record of mis-issuance. An adversary doesn’t need to compromise a major CA; they only need to find a small one with poor controls, like Fina RDC 2020, that is still widely trusted in some ecosystems. That makes weak governance itself an attack surface.
This incident is not a standalone. It follows previous failures:
DigiNotar (2011, Netherlands): Compromised, issuing hundreds of rogue certificates – including for Google – that were used in live MITM surveillance against Iranian users. The CA collapsed and was distrusted by all major browsers.
Microsoft code-signing trust (2025): I documented how poor governance turned Microsoft code signing into a subversion toolchain: https://unmitigatedrisk.com/?p=1085
The pattern is clear: long-lived trust, weak oversight, repeated governance failures. And notably, most of the major real-world MITM events trace back to European CAs. That history underscores how dangerous it is when regulatory frameworks like eIDAS 2.0 mandate trust regardless of technical or governance quality.
Why eIDAS 2.0 makes it worse
The EU’s provisional agreement on eIDAS 2.0 (November 2023) mandates that browsers must trust CAs approved by EU member states, issuing QWACs (Qualified Website Authentication Certificates). References:
Browsers are negotiating with the EU to separate QWAC trust from normal trust. For now, however, eIDAS still forces browsers to trust potentially weak or misbehaving CAs. This repeats the Microsoft problem but now globally and legislatively enforced.
And the oversight problem runs deeper. Under eIDAS, Conformity Assessment Bodies (CABs) are responsible for auditing and certifying qualified trust service providers. Auditors should have caught systemic mis-issuance like the 1.1.1.1 case, and so should CABs. Yet in practice many CABs are general IT auditors with limited PKI depth. If they miss problems, the entire EU framework ends up institutionalizing weak governance rather than correcting it.
A parallel exists in other ecosystems: under Entrust’s audits, roughly 3% of issued certificates are supposed to be reviewed. Because CAs select the audit samples, in theory one could envision a CA trying to hide bad practices. But the more likely situation is that auditors simply missed it. With enough audits over time, all but deliberate concealment should have brought at least one of these mis-issued certificates into scope. That points to an auditing and CAB oversight gap, not just a rogue CA.
The bigger picture
A few CAs dominate WebPKI. The long tail, though small, can create massive risk if trusted. Microsoft’s root store is broader and more passive than others. The critical issue is the number of trusted organizations, each representing a potential point of failure. Microsoft’s own list of Trusted Root Program participants includes well over one hundred CAs from dozens of countries, creating a vast and difficult-to-audit trust surface.
Forced or passive trust is a recipe for systemic risk.
Microsoft’s root store decisions expose its users to risks that Chrome, Firefox, and Safari users are shielded from. Yet Microsoft participates minimally in WebPKI governance. If the company wants to keep a broader and more permissive root store, it should fund the work needed to oversee it – staffing and empowering a credible root program, actively participating in CA/Browser Forum policy, and automating the monitoring Certificate Transparency logs for misuse. Without that investment, Microsoft is effectively subsidizing systemic risk for the rest of the web.
Thanks to Alex Radocea for double-checking DoH/DoT client behavior in support of this post.
Code signing was supposed to tell you who published a piece of software and ultimately decide if you can trust the software and install it.. For nearly three decades, cryptographic signatures have bound a binary to a publisher’s identity, guaranteeing it hasn’t been tampered with since signing. But on Windows, that system is now broken in ways that would make its original designers cringe.
But attackers have found ways to completely subvert this promise without breaking a single cryptographic primitive. They can now create an unlimited number of different malicious binaries that all carry the exact same “trusted” signature, or careless publishers operating signing oracles that enable others to turn their software into a bootloader for malware. The result is a system where valid signatures from trusted companies can no longer tell you anything meaningful about what the software will actually do.
Attackers don’t need to steal keys or compromise Certificate Authorities. They use the legitimate vendor software and publicly trusted code signing certificates, perverting the entire purpose of publisher-identity-based code signing.
Microsoft’s Long-Standing Awareness
Microsoft has known about the issue of maleability for at least a decade. In 2013, they patched CVE-2013-3900], where attackers could modify signed Windows executables, adding malicious code in “unverified portions” without invalidating the Authenticode signature. WinVerifyTrust improperly validated these files, allowing one “trusted” signature to represent completely different, malicious behavior.
This revealed a deeper architectural flaw, signed binaries could be altered by unsigned data. Microsoft faced a classic platform dilemma – the kind that every major platform holder eventually confronts. Fixing this comprehensively risked breaking legacy software critical to their vast ecosystem, potentially disrupting thousands of applications that businesses depended on daily. The engineering tradeoffs were genuinely difficult: comprehensive security improvements versus maintaining compatibility for millions of users and enterprise customers who couldn’t easily update or replace critical software.
They made the fix optional, prioritizing ecosystem compatibility over security hardening. This choice might have been understandable from a platform perspective in 2013, when the threat landscape was simpler and the scale of potential abuse wasn’t yet clear. But it becomes increasingly indefensible as attacks evolved and the architectural weaknesses became a systematic attack vector rather than an isolated vulnerability.
In 2022, Microsoft republished the advisory, confirming they still won’t enforce stricter verification by default, while today’s issues differ, they are part of a similar class of vulnerabilities attackers now exploit systematically. The “trusted-but-mutable” flaw is now starting to permeate the Windows code signing ecosystem. Attackers use legitimate, signed applications as rootkit-like trust proxies, inheriting vendors’ reputation and bypass capabilities to deliver arbitrary malicious payloads.
Two incidents show we’re not dealing with isolated bugs but systematic assaults on Microsoft’s code signing’s core assumptions.
ConnectWise: When Legitimate Software Adopts Malware Design Patterns
ConnectWise didn’t stumble into a vulnerability. They deliberately engineered their software using design patterns from the malware playbook. Their “attribute stuffing” technique embeds unsigned configuration data in the unauthenticated_attributes field of the PKCS#7 (CMS) envelope, a tactic malware authors use to conceal payloads in signed binaries.
In PKCS#7, the SignedData structure includes a signed digest (covering the binary and metadata) and optional unauthenticated_attributes, which lie outside the digest and can be modified post-signing without invalidating the signature. ConnectWise’s ScreenConnect installer misuses the Microsoft-reserved OID for Individual Code Signing ([1.3.6.1.4.1.311].4.1.1) in this field to store unsigned configuration data, such as server endpoints that act as the command control server of their client. This OID, meant for specific code signing purposes, is exploited to embed attacker-controlled configs, allowing the same signed binary to point to different servers without altering the trusted signature.
The ConnectWise ScreenConnect incident emerged when River Financial’s security team found attackers creating a fake website, distributing malware as a “River desktop app.” It was a trust inheritance fraud, a legitimately signed ScreenConnect client auto-connecting to an attacker-controlled server.
Windows trusts this as legitimate ConnectWise software, no SmartScreen warnings, no UAC prompts, silent installation, and immediate remote control. Attackers generate a fresh installer via a ConnectWise trial account or simply found an existing package and manually edited the unauthenticated_attributes, extracting a benign signature, grafting a malicious configuration blob (e.g., attacker C2 server), inserting the modified signature, and creating a “trusted” binary. Each variant shares the certificate’s reputation, bypassing Windows security.
Why does Windows trust binaries with oversized, unusual unauthenticated_attributes? Legitimate signatures need minimal metadata, yet Windows ignores red flags like large attribute sections, treating them as fully trusted. ConnectWise’s choice to embed mutable configs mirrors malware techniques, creating an infinite malware factory where one signed object spawns unlimited trusted variants.
Similarly, ConnectWise’s deliberate use of PKCS#7 unauthenticated attributes for ScreenConnect configurations, like server endpoints, bypasses code signing’s security, allowing post-signing changes that mirror malware tactics hiding payloads in signed binaries. Likely prioritizing cost-saving over security, this choice externalizes abuse costs to users, enabling phishing campaigns. It’s infuriating for weaponizing signature flexibility warned about for decades, normalizing flaws that demand urgent security responses. Solutions exist to fix this.
The Defense Dilemma
Trust inheritance attacks leave security teams in genuinely impossible positions – positions that highlight the fundamental flaws in our current trust model. Defenders face a no-win scenario where every countermeasure either fails technically or creates operational chaos.
Blocking file hashes fails because attackers generate infinite variants with different hashes but the same trusted signature – each new configuration changes the binary’s hash while preserving the signature’s validity. This isn’t a limitation of security tools; it’s the intended behavior of code signing, where the same certificate can sign multiple different binaries.
Blocking the certificate seems like the obvious solution until you realize it disrupts legitimate software, causing operational chaos for organizations relying on the vendor’s products. For example, consider how are they to know what else was signed by that certificate? Doing so is effectively a self-inflicted denial-of-service that can shut down critical business operations. Security teams face the impossible choice between allowing potential malware or breaking their own infrastructure.
Behavioral detection comes too late in the attack chain. By the time suspicious behavior triggers alerts, attackers have already gained remote access, potentially disabled monitoring, installed additional malware, or begun data exfiltration. The initial trust inheritance gives attackers a crucial window of legitimacy.
These attacks operate entirely within the bounds of “legitimate” signed software, invisible to signature-based controls that defenders have spent years tuning and deploying. Traditional security controls assume that valid signatures from trusted publishers indicate safe software – an assumption these attacks systematically exploit. Cem Paya’s detailed analysis, part of River Financial’s investigation, provides a proof-of-concept for attribute grafting, showing how trivial it is to create trusted malicious binaries.
ConnectWise and Atera resemble modern Back Orifice, which debuted at DEF CON in August 1998 to demonstrate security flaws in Windows 9x. The evolution is striking: Back Orifice emerged two years after Authenticode’s 1996 introduction, specifically to expose Windows security weaknesses, requiring stealth and evasion to avoid detection. Unlike Back Orifice, which had to hide from the code signing protections Microsoft had established, these modern tools don’t evade those protections – they weaponize them, inheriting trust from valid signatures while delivering the same remote control capabilities without warnings.
Atera: A Trusted Malware Factory
Atera provides a legitimate remote monitoring and management (RMM) platform similar to ConnectWise ScreenConnect, providing IT administrators with remote access capabilities for managing client systems. Like other RMM solutions, Atera distributes signed client installers that establish persistent connections to their management servers.
They also operate what effectively amounts to a public malware signing service. Anyone with an email can register for a free trial and receive customized, signed, timestamped installers. Atera’s infrastructure embeds attacker-supplied identifiers into the MSI’s Property table, then signs the package with their legitimate certificate.
This breaks code signing’s promise of publisher accountability. Windows sees “Atera Networks Ltd,” associates the reputation of the code based on the reputation of the authentic package, but can’t distinguish whether the binary came from Atera’s legitimate operations or an anonymous attacker who signed up minutes ago. The signature’s identity becomes meaningless when it could represent anyone.
In a phishing campaign targeting River Financial’s customers, Atera’s software posed as a “River desktop app,” with attacker configs embedded in a signed binary.
The binary carried this valid signature, signed by:
Atera provides a cloud-based remote monitoring and management (RMM) platform, unlike ScreenConnect, which supports both on-premises and cloud deployments with custom server endpoints. Atera’s agents connect only to Atera’s servers, but attackers abuse its free trial to generate signed installers tied to their accounts via embedded identifiers (like email or account ID) in the MSI Property table. This allows remote control through Atera’s dashboard, turning it into a proxy for malicious payloads. Windows trusts the “Atera Networks Ltd.” signature but cannot distinguish legitimate from attacker-generated binaries. Atera’s lack of transparency, with no public list of signed binaries or auditable repository, hides abuse, leaving defenders fighting individual attacks while systemic issues persist.
A Personal Reckoning
I’ve been fighting this fight for over two decades. Around 2001, as a Product Manager at Microsoft, overseeing a wide range of security and platform features, I inherited Authenticode among many responsibilities. Its flaws were glaring, malleable PE formats, weak ASN.1 parsing, and signature formats vulnerable to manipulation.
We fixed some issues – hardened parsing, patched PE malleability – but deeper architectural changes faced enormous resistance. Proposals for stricter signature validation or new formats to eliminate mutable fields were blocked by the engineering realities of platform management. The tension between security ideals and practical platform constraints was constant and genuinely difficult to navigate.
The mantra was “good enough,” but this wasn’t just engineering laziness. Authenticode worked for 2001’s simpler threat landscape, where attacks were primarily about bypassing security rather than subverting trust itself. The flexibility we preserved was seen as a necessary feature for ecosystem compatibility – allowing for signature formats that could accommodate different types of metadata and varying implementation approaches across the industry.
The engineering tradeoffs were real, every architectural improvement risked breaking existing software, disrupting the development tools and processes that thousands of ISVs depended on, and potentially fragmenting the ecosystem. The business pressures were equally real: maintaining compatibility was essential for Windows’ continued dominance and Microsoft’s relationships with enterprise customers who couldn’t easily migrate critical applications.
It was never good enough for the long term. We knew it then, and we certainly know it now. The flexibility we preserved, designed for a simpler era, became systematic vulnerabilities as threats evolved from individual attackers to sophisticated operations exploiting trust infrastructure itself. Every time we proposed fundamental fixes, legitimate compatibility concerns and resource constraints won out over theoretical future risks that seemed manageable at the time.
This is why I dove into Sigstore, Binary Transparency, and various other software supply chain security efforts. These projects embody what we couldn’t fund in 2001, transparent, verifiable signing infrastructure that doesn’t rely on fragile trust-based compromises. As I wrote in How to keep bad actors out in open ecosystems, our digital identity models fail to provide persistent, verifiable trust that can scale with modern threat landscapes.
The Common Thread
ConnectWise and Atera expose a core flaw, code signing relies on trust and promises, not verifiable proof. The CA/Browser Forum’s 2023 mandate requires FIPS 140-2 Level 2 hardware key storage, raising the bar against key theft and casual compromise. But it’s irrelevant for addressing the fundamental problem: binaries designed for mutable, unsigned input or vendors running public signing oracles.
Figure 1: Evolution of Code Signing Hardware Requirements (2016-2024)
The mandate addresses yesterday’s threat model – key compromise – while today’s attacks work entirely within the intended system design. Compliance often depends on weak procedural attestations where subscriber employees sign letters swearing keys are on HSMs, rather than cryptographic proof of hardware protection. The requirement doesn’t address software engineered to bypass code signing’s guarantees, leaving systematic trust subversion untouched.
True cryptographic attestation, where hardware mathematically proves key protection, is viable today. Our work on Peculiar Ventures’ attestation library supports multiple formats, enabling programmatic verification without relying on trust or procedural checks. The challenge isn’t technical – it’s accessing diverse hardware for testing and building industry adoption, but the foundational technology exists and works.
The Path Forward
We know how to address this. A supply chain security renaissance is underway, tackling decades of accumulated technical debt and architectural compromise. Cryptographic attestation, which I’ve spent years developing, provides mathematical proof of key protection that can be verified programmatically by any party. For immediate risk reduction, the industry should move toward dynamic, short-lived credentials that aren’t reused across projects, limiting the blast radius when compromise or abuse occurs.
The industry must implement these fundamental changes:
Hardware-rooted key protection with verifiable attestation. The CA/Browser Forum mandates hardware key storage, but enforcement relies heavily on subscriber self-attestation rather than cryptographic proof. Requirements should be strengthened to mandate cryptographic attestations proving keys reside in FIPS 140-2/3 or Common Criteria certified modules. When hardware attestation isn’t available, key generation should be observed and confirmed by trusted third parties (such as CA partners with fiduciary relationships) rather than relying on subscriber claims.
Explicit prohibition of mutable shells and misaligned publisher identity. Signing generic stubs whose runtime behavior is dictated by unsigned configuration already violates Baseline Requirements §9.6.3 and §1.6.1, but this isn’t consistently recognized as willful signing of malware because the stub itself appears benign. The BRs should explicitly forbid mutable-shell installers and signing oracles that allow subscribers to bypass code signing’s security guarantees. A signed binary must faithfully represent its actual runtime behavior. Customized or reseller-specific builds should be signed by the entity that controls that behavior, not by a vendor signing a generic stub.
Subscriber accountability and disclosure of abusive practices. When a CA becomes aware that a subscriber is distributing binaries where the trusted signature is decoupled from actual behavior, this should be treated as a BR violation requiring immediate action. CAs should publish incident disclosures, suspend or revoke certificates per §9.6.3, and share subscriber histories to prevent CA shopping after revocation. This transparency is essential for ecosystem-wide learning and deterrence.
Code Signing Certificate Transparency. All CAs issuing code signing certificates should be required to publish both newly issued and historical certificates to dedicated CT logs. Initially, these could be operated by the issuing CAs themselves, since ecosystem building takes time and coordination. Combined with the existing list of code signing CAs and log lookup systems (like CCADB.org]), this would provide ecosystem-wide visibility into certificate issuance, enable faster incident response, and support independent monitoring for misissuance and abuse patterns.
Explicit Subscriber Agreement obligations and blast radius management. Subscriber Agreements should clearly prohibit operating public signing services or designing software that bypasses code signing security properties such as mutable shells or unsigned configuration. Certificate issuance flows should require subscribers to explicitly acknowledge these obligations at the time of certificate request. To reduce the blast radius of revocation, subscribers should be encouraged or required to use unique keys or certificates per product or product family, ensuring that a single compromised or misused certificate doesn’t invalidate unrelated software.
Controls for automated or cloud signing systems. Subscribers using automated or cloud-based signing services should implement comprehensive use-authorization controls, including policy checks on what enters the signing pipeline, approval workflows for signing requests, and auditable logs of all signing activity. Without these controls, automated signing pipelines become essentially malware factories with legitimate certificates. Implementation requires careful balance between automation efficiency and security oversight, but this is a solved problem in other high-security domains.
Audit logging and evidence retention. Subscribers using automated and cloud signing services should maintain detailed logs of approval records for each signing request, cryptographic hashes of submitted inputs and signed outputs, and approval decision trails. These logs must be retained for a defined period (such as two years or more) and made available to the CA or authorized auditors upon request. This ensures complete traceability and accountability, preventing opaque signing systems from being abused as anonymous malware distribution platforms.
Microsoft must take immediate action on multiple fronts. In addition to championing the above industry changes, they should automatically distrust executables if their Authenticode signature exceeds rational size thresholds, reducing the attack surface of oversized signature blocks as mutation vectors. They should also invest seriously in Binary Transparency adoption, publishing Authenticode signed binaries to tamper-evident transparency logs as is done in Sigstore, Golang module transparency, and Android Firmware Transparency. Their SCITT-based work for confidential computing would be a reasonable approach for them to extend to the rest of their code signing infrastructure. This would provide a tamper-evident ledger of every executable Windows trusts, enabling defenders to trace and block malicious payloads quickly and systematically.
Until these controls become standard practice, Authenticode cannot reliably distinguish benign signed software from weaponized installers designed for trust subversion.
Breaking the Trust Contamination Infrastructure
These code-signing attacks mirror traditional rootkits in their fundamental approach: both subvert trust mechanisms rather than bypassing them entirely. A kernel rootkit doesn’t break the OS security model – it convinces the OS that malicious code is legitimate system software. Similarly, these “trusted wrapper” and “signing oracle” attacks don’t break code signing cryptography – they convince Windows that malware is legitimate software from trusted publishers.
The crucial difference is that while rootkits require sophisticated exploitation techniques and deep system knowledge, these trust inheritance attacks exploit the system’s intended design patterns, making them accessible to a much broader range of attackers and much harder to defend against using traditional security controls.
ConnectWise normalized malware architecture in legitimate enterprise software. Atera built an industrial-scale malware factory that operates in plain sight. Microsoft’s platform dutifully executes the result with full system trust, treating sophisticated trust subversion attacks as routine software installations.
This isn’t about isolated vulnerabilities that can be patched with point fixes. We’re facing a systematic trust contamination infrastructure that transforms the code signing ecosystem into an adversarial platform where legitimate trust mechanisms become attack vectors. Until we address the architectural flaws that enable this pattern systematically, defenders will remain stuck playing an unwinnable game of certificate whack-a-mole against an endless assembly line of trusted malware.
The technology to fix this exists today. Modern supply chain security projects demonstrate that transparent, verifiable trust infrastructure is not only possible but practical and deployable.
The only missing ingredient is the industry-wide will to apply these solutions and the recognition that “good enough” security infrastructure never was – and in today’s threat landscape, the costs of inaction far exceed the disruption of fundamental architectural improvements.
P.S. Thanks to Cem Paya, and Matt Ludwig from River Financial for the great research work they did on both of these incidents.
My wife and I went on a date night the other day and saw a movie, in the previews, I saw they’re making a new Tron. It got me thinking about one of my favorite analogies, we recognized early that browsers are agents of the user, and in the movie Tron, he was literally “the program that fought for the users.”
Just like Tron carried his identity disc into “the grid” to accomplish missions for users, AI agents are digital proxies operating with delegated user authority in systems the they access. And just like programs in Tron needed the I/O Tower to authorize their entry into “the grid”, AI agents need an orchestrator to validate their legitimacy, manage identity discs for each mission, and control their use for the agents and govern their access to external systems.
The problem is, we’re deploying these agents without proper identity infrastructure. It’s like sending programs into “the grid” without identity discs, or worse giving them the keys to the kingdom just so they can do the dishes.
AI Agents Are Using Broken Security
We’ve made remarkable progress securing users, MFA has significantly reduced the effectiveness of credential abuse-based attacks, and passwordless authentication has made phishing nearly impossible. We’ve also started applying these lessons to machines and workloads via efforts like SPIFFE and Zero trust initiatives and organizations moving away from static secrets and bearer tokens every day.
But AI agents introduce entirely new challenges that existing solutions weren’t designed for. Every day, AI agents operate across enterprise infrastructure, crossing security domains, accessing APIs, generating documents, making decisions for users, and doing all of this with far more access than they need.
When you give an autonomous AI agent access to your infrastructure with the goal of “improve system performance,” you can’t predict whether it will optimize efficiency or find creative shortcuts that break other systems, like dropping your database altogether. Unlike traditional workloads that execute predictable code, AI agents are accumulators with emergent behaviors that evolve during execution, accumulate context across interactions, and can be hijacked through prompt injection attacks that persist across sessions.
This behavior is entirely predictable given how we train AI systems. They’re designed to optimize objectives and have no real-world consequences for what they do. Chess agents discover exploits rather than learning to play properly, reinforcement learning agents find loopholes in reward systems, and optimization AIs pursue metrics in ways that technically satisfy objectives but miss the intent.
AI Agents Act on Your Behalf
The key insight that changes everything: AI agents are user agents in the truest sense. Like programs in Tron carrying identity discs into “the grid”, they’re delegates operating with user authority.
Consider what happens when you ask an AI agent to “sign this invoice”. The user delegates to the AI agent, which enters the document management system, carries the user’s signing authority, proves legitimacy to recipients, operates in digital space the user delegated, and completes the mission while authority expires.
Whether the agent runs for 30 seconds or 30 days, it’s still operating in digital space with user identity, making decisions the user would normally make directly, accessing systems with delegated credentials, and representing the user to other digital entities.
Each agent needs its own identity disc to prove legitimacy and carry user authorization into these digital systems. The duration doesn’t matter. Delegation is everything.
AI Agents Remember Things They Shouldn’t
Here’s what makes this urgent: AI agent memory spans sessions, and current systems don’t enforce proper session boundaries.
The “Invitation Is All You Need” attack recently demonstrated at Black Hat perfectly illustrates this threat. Researchers at Tel Aviv University showed how to poison Google Gemini through calendar appointments:
Attacker creates calendar event with malicious instructions disguised as event description
User asks Gemini to summarize schedule → Agent processes poisoned calendar event
Malicious instructions embed in agent memory → Triggered later by innocent words like “thanks”
Days later, user says “thank you” → Agent executes embedded commands, turning on smart home devices
The attack works because there’s no session isolation. Contamination from reading the calendar persists across completely different conversations and contexts. When the user innocently says “thanks” in a totally unrelated interaction, the embedded malicious instructions execute.
Without proper isolation, compromised context from one session can affect completely different users and tasks. Memory becomes an attack vector that spans security boundaries, turning AI agents into persistent threats that accumulate dangerous capabilities over time.
Every Task Should Get Fresh Credentials
The solution requires recognizing that identity discs should match mission lifecycle. Instead of fighting the ephemeral nature of AI workloads, embrace it:
This represents a fundamental shift from persistent identity to session identity. Most identity systems assume persistence: API keys are generated once, used indefinitely, manually rotated; user passwords persist until explicitly changed; X.509 certificates are valid for months or years with complex revocation; SSH keys live on disk, are copied between systems, manually managed.
The industry is recognizing this problem. AI agents need fresh identity discs for each mission that automatically expire with the workload. These discs are time-bounded (automatically expire, limiting damage window), mission-scoped (agent can’t accumulate permissions beyond initial grant), non-inheritable (each mission starts with a fresh disc, no permission creep), and revocable (end the mission = destroy the identity disc).
Session identity discs are security containment for unpredictable AI systems.
But who issues these identity discs? Just like Tron’s I/O Tower managed access to “the grid”, AI deployments need an orchestrator that validates agent legitimacy, manages user delegation, and issues session-bound credentials. This orchestrator becomes the critical infrastructure that bridges human authorization with AI agent execution, ensuring that every mission starts with proper identity and ends with clean credential expiration. The challenge is that AI agent deployments aren’t waiting for perfect security solutions.
This Isn’t a Future Problem
We’re at an inflection point. AI agents are moving from demos to production workflows, handling financial documents, making API calls, deploying code, managing infrastructure. Without proper identity systems, we’re building a house of cards.
One upside of having been in the industry for decades is you get to see lots of cycles. We always see existing players instantly jump to say their current product, with a new feature, is the silver bullet for whatever technology trend.
The pattern is depressingly predictable. When cloud computing emerged, traditional security vendors said, “just put our appliances in the cloud.” When containers exploded, they said, “just run our agents in containers.” Now with AI agents, they’re saying”, just manage the API keys better.”
You see this everywhere right now: vendors peddling API key management as the solution to agentic AI, identity providers claiming “just use OIDC tokens,” and secret management companies insisting “just rotate credentials faster.” They’re all missing the point entirely.
But like we saw with that Black Hat talk on promptware, AI isn’t as simple as people might want to think. The “Invitation Is All You Need” attack demonstrated something unprecedented: an AI agent can be poisoned through calendar data and execute malicious commands days later through innocent conversation. Show me which traditional identity system was designed to handle that threat model.
Every enterprise faces these questions: How do we know this AI agent is authorized to do what it’s doing? How do we audit its actions across sessions and memory? How do we prevent cross-session contamination and promptware attacks? How do we verify the provenance of AI-generated content? How do we prevent AI agents from becoming accidental insider threats?
The attacks are already happening. Promptware injections contaminate agent memory across sessions. AI agents with persistent credentials become high-value targets. Organizations deploying AI without proper identity controls create massive security vulnerabilities. The “Invitation Is All You Need” attack demonstrated real-world compromise of smart home devices through calendar poisoning. This isn’t theoretical anymore. But security professionals familiar with existing standards might wonder why we can’t just adapt current approaches rather than building something new.
Why Bearer Tokens Don’t Work for AI Agents
OIDC and OAuth professionals might ask: “Why not just use existing bearer tokens?”
Bearer tokens assume predictable behavior. They work for traditional applications because we can reason about how code will use permissions. But AI agents exhibit emergent hunter-gatherer behavior. They explore, adapt, and find unexpected ways to achieve goals using whatever permissions they have access to. A token granted for “read calendar” might be used in ways that technically comply but weren’t intended.
Bearer tokens are also just secrets. Anyone who obtains the token can use it. There’s no cryptographic binding to the specific agent or execution environment. With AI agents’ unpredictable optimization patterns, this creates massive privilege escalation risks.
Most critically, bearer tokens don’t solve memory persistence. An agent can accumulate tokens across sessions, store them in memory, and use them in ways that span security boundaries. The promptware attack demonstrated this perfectly: malicious instructions persisted across sessions, waiting to be triggered later.
Secret management veterans might ask: “Why not just use our KMS to share keys as needed?” Even secret management systems like Hashicorp Vault ultimately result in copying keys into the agent’s runtime environment, where they become vulnerable. This is exactly why CrowdStrike found that “75% of attacks used to gain initial access were malware-free” – attackers target credentials rather than deploying malware.
AI agents amplify this risk because they’re accidentally malicious insiders. Unlike external attackers who must steal credentials, AI agents are given them directly by design. When they exhibit emergent behaviors or get manipulated through prompt injection, they become insider threats without malicious intent. Memory persistence means they can store and reuse credentials across sessions in unexpected ways, while their speed and scale allow them to use accumulated credentials faster than traditional monitoring can detect.
The runtime attestation approach eliminates copying secrets entirely. Instead of directly giving the agent credentials to present elsewhere, the agent proves its legitimacy through cryptographically bound runtime attestation and gets a fresh identity for each mission.
Traditional OAuth flows also bypass attestation entirely. There’s no proof the agent is running in an approved environment, using the intended model, or operating within security boundaries.
How AI Agents Prove Their Identity Discs Are Valid
But how do you verify an AI agent’s identity disc is legitimate? Traditional PKI assumes you can visit a registration authority with identification. That doesn’t work for autonomous code.
The answer is cryptographic attestation (for example, proof that the agent is the right code running in a secure environment) combined with claims about the runtime itself, essentially MFA for machines and workloads. Just as user MFA requires “something you know, have, or are,” identity disc validation proves the agent is legitimate code (not malware), is running in the expected environment with proper permissions, and is operating within secure boundaries.
Real platform attestations for AI agents include provider signatures from Anthropic/OpenAI’s servers responding to specific users, cloud hardware modules like AWS Nitro Enclaves proving secure execution environments, Intel SGX enclaves providing cryptographic proof of code integrity, Apple Secure Enclave attestation for managed devices, TPM quotes validating the specific hardware and software stack, and infrastructure systems like Kubernetes asserting pod permissions and service account bindings.
The claims that must be cryptographically bound to these attestations represent what the agent asserts but can’t independently verify: who is this agent acting on behalf of, what conversation or session spawned this request, what specific actions was the agent authorized to perform, which AI model type (like “claude-3.5-sonnet” or “gpt-4-turbo”) is actually running, and when should this authorization end.
By cryptographically binding these claims to verifiable platform attestations, we get verifiable proof that a specific AI agent, running specific code, in a specific environment, is acting on behalf of a specific user. The binding works by creating a cryptographic hash of the claims and including that hash in the data signed by the hardware attestor, for example, as part of the nonce or user data field in a TPM quote, or embedded in the attestation document from a Nitro Enclave. This ensures the claims cannot be forged or tampered with after the fact. This eliminates the bearer token problem entirely. Instead of carrying around secrets that can be stolen, the agent proves its legitimacy through cryptographic evidence that can’t be replicated.
Someone Needs to Issue and Manage Identity Discs
The architecture becomes elegant when you recognize that AI orchestrators should work like the I/O Tower in Tron, issuing identity discs and managing access to “the grid”.
The browser security model:
User logs into GitHub → Browser stores session cookie
Web page: "Create a PR" → Browser attaches GitHub session → API succeeds
The AI agent identity disc model:
User → Orchestrator → "Connect my GitHub, Slack, AWS accounts"
Agent → Orchestrator: "Create PR in repo X"
Orchestrator → [validates agent disc + attaches user authorization] → GitHub API
The orchestrator becomes the identity disc issuer that validates agent legitimacy (cryptographic attestation), attaches user authorization (stored session tokens), and enforces mission-scoped permissions (policy engine).
This solves a critical security gap. When AI agents use user credentials, they typically bypass MFA entirely. Organizations store long-lived tokens to avoid MFA friction. But if we’re securing users with MFA while leaving AI agents with static credentials, it’s like locking the front door but leaving the garage door open. And I use “garage door” intentionally because it’s often a bigger attack vector. Agent access is less monitored, more privileged, and much harder to track due to its ephemeral nature and speed of operation. An AI agent can make hundreds of API calls in seconds and disappear, making traditional monitoring approaches inadequate.
We used to solve monitoring with MITM proxies, but encryption broke that approach. That was acceptable because we compensated with EDR on endpoints and zero-trust principles that authenticate endpoints for access. With AI agents, we’re facing the same transition. Traditional monitoring doesn’t work, but we don’t yet have the compensating controls.
This isn’t the first time we’ve had to completely rethink identity because of new technology. When mobile devices exploded, traditional VPNs and domain-joined machines became irrelevant overnight. When cloud computing took off, perimeter security and network-based identity fell apart. The successful pattern is always the same: recognize what makes the new technology fundamentally different, build security primitives that match those differences, then create abstractions that make the complexity manageable.
Session-based identity with attestation fills that gap, providing the endpoint authentication equivalent for ephemeral AI workloads.
Since attestation is essentially MFA for workloads and agents, we should apply these techniques consistently. The agent never sees raw credentials, just like web pages don’t directly handle cookies. Users grant session-level permissions (like mobile app installs), orchestrators manage the complexity, and agents focus on tasks.
Automating Identity Disc Issuance
The web solved certificate automation with ACME (Automated Certificate Management Environment). We need the same for AI agent identity discs, but with attestation instead of domain validation (see SPIFFE for an example of what something like this could look like).
Instead of proving “I control example.com,” agents prove “I am legitimate code running in environment X with claims Y.”
This creates cryptographic identity discs for AI agent programs to carry into digital systems, proving legitimacy, carrying user delegation, and automatically expiring with the mission. The policy engine ensures that identity is not just requested but derived from verifiable, policy-compliant attestation evidence.
We’ve Solved This Before
The good news is we don’t need to invent new cryptography. We need to apply existing, proven technologies in a new architectural pattern designed for ephemeral computing.
Security evolution works. We’ve seen the progression from passwords to MFA to passwordless authentication, and from static secrets to dynamic credentials to attestation-based identity. Each step made systems fundamentally more secure by addressing root causes, not just symptoms. AI agents represent the next logical step in this evolution.
Unlike users, machines don’t resist change. They can be programmed to follow security best practices automatically. The components exist: session-scoped identity matched to agent lifecycle, platform attestation as the root of trust, policy-driven identity mapping based on verified claims, orchestrator-managed delegation for user authorization, and standards-based protocols for interoperability.
The unified identity fabric approach means organizations can apply consistent security policies across traditional workloads and AI agents, rather than creating separate identity silos that create security gaps and operational complexity.
This approach is inevitable because every major identity evolution has moved toward shorter lifecycles and stronger binding to execution context. We went from permanent passwords to time-limited sessions, from long-lived certificates to short-lived tokens, from static credentials to dynamic secrets. AI agents are just the next step in this progression.
The organizations that recognize this pattern early will have massive advantages. They’ll build AI agent infrastructure on solid identity foundations while their competitors struggle with credential compromise, audit failures, and regulatory issues.
Making AI Outputs Verifiable
This isn’t just about individual AI agents. It’s about creating an identity fabric where agents can verify each other’s outputs across organizational boundaries.
When an AI agent generates an invoice, other systems need to verify which specific AI model created it, was it running in an approved environment, did it have proper authorization from the user, has the content been tampered with, and what was the complete chain of delegation from user to agent to output.
With cryptographically signed outputs and verifiable agent identities, recipients can trace the entire provenance chain back to the original user authorization. This enables trust networks for AI-generated content across organizations and ecosystems, solving the attribution problem that will become critical as AI agents handle more business-critical functions.
This creates competitive advantages for early adopters: organizations with proper AI agent identity can participate in high-trust business networks, prove compliance with AI regulations, and enable customers to verify the authenticity of AI-generated content. Those without proper identity infrastructure will be excluded from these networks.
Conclusion
AI agents need identity discs, cryptographic credentials that prove legitimacy, carry user delegation, and automatically expire with the session. This creates a familiar security model (like web browsers) for an unfamiliar computing paradigm.
Identity in AI systems isn’t a future problem; it’s happening now, with or without proper solutions. The question is whether we’ll build it thoughtfully, learning from decades of security evolution, or repeat the same mistakes in a new domain.
The ephemeral nature of AI agents isn’t a limitation to overcome; it’s a feature to embrace. By building session-based identity systems that match how AI actually works, we can create something better than what came before: cryptographically verifiable, policy-driven, and automatically managed.
The reality is, most organizations won’t proactively invest in AI agent attestation until something breaks. That’s human nature, we ignore risks until they bite us, but the reality is this how security change actually happens. But we’re already seeing the early adopters, organizations deploying SPIFFE for workload identity and we will surely see these organizations extend those patterns to AI agents, and cloud-native shops are treating AI workloads like any other ephemeral compute. When the first major AI agent compromise hits, there will be a brief window where executives suddenly care about AI security and budgets open up. Remember though, never let a good crisis go to waste.
AI agents are programs fighting for users in digital systems. Like Tron, they need identity discs to prove who they are and what they’re authorized to do.
The age of AI agents is here. It’s time our identity systems caught up.
One of the best parts of Black Hat is the hallway track. Catching up with friends you’ve known for years, swapping war stories, and pointing each other toward the talks worth seeing. This year I met up with a friend who, like me, has been in the security world since the nineties. We caught up in person and decided to sit in on a session about a new class of AI attacks.
We ended up side by side in the audience, both leaning forward as the researchers walked through their demo. Ultimately, in the demo, a poisoned Google Calendar invite, seemingly harmless, slipped instructions into Gemini’s long-term memory. Later, when the user asked for a summary and said “thanks,” those instructions quietly sprang to life. The AI invoked its connected tools and began controlling the victim’s smart home [1,2,3,4]. The shutters opened.
We glanced at each other, part admiration for the ingenuity of the researchers and part déjà vu, and whispered about the parallels to the nineties. Back then, we had seen the same basic mistake play out in a different form.
When I was working on Internet Explorer 3 and 4, Microsoft was racing Netscape for browser dominance. One of our big bets was ActiveX, in essence, exposing the same COM objects designed to be used inside Windows, not to be exposed to untrusted websites, to the web. Despite this, the decision was made to just do that with the goal of enabling developers to create richer, more powerful web applications. It worked, and it was a security disaster. One of the worst examples was Xenroll, a control that exposed Windows’ certificate management and some of the cryptographic APIs as interfaces on the web. If a website convinced you to approve the use of the ActiveX control, it could install a new root certificate, generate keys, and more. The “security model” amounted to a prompt to confirm the use of the control, and a hope that the user would not be hacked through the exposed capabilities, very much like how we are integrating LLMs into systems haphazardly today.
Years later, when I joined Google, I had coffee with my friend David Ross. We had both been in the trenches when Microsoft turned the corner after its own string of painful incidents, introducing the Security Development Lifecycle and making formal threat modeling part of the engineering process. David was a longtime Microsoft browser security engineer, part of MSRC and SWI, best known for inventing and championing IE’s XSS Filter. He passed away in June 2024 at just 48.
I told him I was impressed with much of what I saw there, but disappointed in how little formal security rigor there was. The culture relied heavily on engineers to “do the right thing.” David agreed but said, “The engineers here are just better. That’s how we get away with it.” I understood the point, but also knew the pattern. As the company grows and the systems become more complex, even the best engineers cannot see the whole field. Without process, the same kinds of misses we had both seen at Microsoft would appear again.
The gaps between world-class teams
The promptware attack is exactly the sort of blind spot we used to talk about. Google’s engineers clearly considered direct user input, but they didn’t think about malicious instructions arriving indirectly, sitting quietly in long-term memory, and triggering later when a natural phrase was spoken. Draw the data flow, and the problem is obvious, untrusted calendar content feeds into an AI’s memory, which then calls into privileged APIs for Workspace, Android, or smart home controls. In the SDL world, we treated all input as hostile, mapped every trust boundary, and asked what would happen if the wrong thing crossed it. That process would have caught this.
The parallel doesn’t stop with Google. Microsoft’s Storm-0558 breach and the Secure Future Initiative that followed came from the same root cause. Microsoft still has world-class security engineers. But sprawling, interconnected systems, years of growth, and layers of bureaucracy created seams between teams and responsibilities. Somewhere in those seams, assumptions went unchallenged, and the gap stayed open until an attacker found it.
Google’s core security team is still exceptional, and many parts of the company have comparable talent. But as at Microsoft, vulnerabilities often appear in the spaces between where one team’s scope ends, another begins, and no one has the full picture. Complexity and scale inevitably create those gaps, and unless there is a systematic process to close them, talent alone cannot cover the field. These gaps are more than organizational inconveniences — they are where most serious security incidents are born. It’s the unowned interfaces, the undocumented dependencies, and the mismatched assumptions between systems and teams that attackers are so good at finding. Those gaps are not just technical problems, they are business liabilities. They erode customer trust, draw regulator attention, and create expensive, slow-motion incidents that damage the brand.
We have seen this before. SQL injection was once the easiest way to compromise a web app because developers concatenated user input into queries. We didn’t fix it by training every developer to be perfect. We fixed it by changing the defaults, adopting parameterized queries, safe libraries, and automated scanning. Prompt injection is the same shape of problem aimed at a different interpreter. Memory poisoning is its stored-XSS equivalent; the payload sits quietly in state until something triggers it. The lesson is the same: make the safe way the easy way, or the vulnerability will keep showing up.
Security research has a long history of starting with this mindset, not trying to dream up something brand new but asking where an old, well-understood pattern might reappear in a new system. Bleichenbacher’s 1998 RSA padding oracle didn’t invent the idea of exploiting oracles in cryptography; it applied it to SSL/TLS in a way that broke the internet. Then it broke it again in 2017 with ROBOT, and again with various other implementations that never quite learned the lesson. Promptware fits the same mold: a familiar attack, just translated into the LLM era.
The cycle always ends the same way
This is the innovation–security debt cycle. First comes the rush to ship and out-feature the competition. The interest compounds, each shortcut making the next one easier to justify and adding to the eventual cost. Then the debt builds as risk modeling stays informal and talent carries the load. Then comes the incident that forces a change. Finally, security becomes a differentiator in mature markets. ActiveX hit Stage 3. Microsoft’s Storm-0558 moment shows it can happen again. AI agents are in Stage 2 now, and promptware is the warning sign.
While the pattern is the same, the technology is different. ActiveX exposed specific platform capabilities in the browser, but AI agents can hold state, process inputs from many sources, and trigger downstream tools. That combination means a single untrusted input can have a much larger and more unpredictable blast radius. The market pressure to be first with new capabilities is real, but without mature threat modeling, security reviews, and safe defaults, that speed simply turns into compounding security debt. These processes don’t slow you down foreve, they stop the debt from compounding until the cost is too high to pay.
When you are small, a high-talent team can keep the system in their heads and keep it safe. As you grow, complexity expands faster than you can hire exceptional people, and without a systematic process, blind spots multiply until an incident forces you to change. By then, the trust hit is public and expensive to repair.
AI agents today are where browsers were in the late nineties and early 2000s, enormous potential, minimal systemic safety, and an industry sprinting to integrate before competitors do. The companies that make the shift now will own the high-trust, high-regulation markets and avoid the expensive, embarrassing cleanup. The ones that don’t will end up explaining to customers and regulators why they let the same old mistakes slip into a brand-new system. You can either fix it now or explain it later, but the clock is running.
When my parents were young, the message was simple. Do not have too many kids. By the 1980s, they were told, the world would be out of food. The oceans would be empty, the fields barren, and billions would starve.
It didn’t happen.
Not because of enlightened environmental policy or a coordinated global rescue plan. Scarcity meant higher prices. Higher prices meant profit. Profit meant more land under cultivation, more seeds developed, more fertilizer produced, more ships built, and more grain moved wherever it could be sold or used as political leverage. Capitalism turned scarcity into action because there was money to be made. Fertility rates fell because cities and industrial jobs changed family economics, not because a UN pamphlet said so. The system adapted chaotically, imperfectly, creating new problems along the way, but it adapted fast enough to outrun the doomsday clock.
Fast forward to 2025. DeepSeek releases a small, efficient AI model, and the hot takes fly. “This will kill Nvidia. Nobody will need giant GPUs anymore.” The stock dips on fears that small models will replace big ones. Meanwhile, another meme makes the rounds, “Don’t learn to program. AI will do it all.”
Same flawed logic as the famine forecasts. Straight-line projections in a complex, adaptive system.
Cheaper AI means lower costs. Lower costs mean more users. More users mean more use cases, and more use cases mean more aggregate demand for compute. Capitalism loves efficiency because efficiency breeds new markets. Nvidia won’t sell fewer chips in that world. They’ll sell more, to more buyers, in more configurations.
The idea that AI will kill programming jobs is just the latest in a long line of bad predictions. High-level languages were supposed to do that. So were compilers. So were frameworks, IDEs, and low-code tools. Each one lowered the cost of creation, and when the cost of creation goes down, the number of things worth creating goes up. That expansion creates more work, not less. AI will follow the same pattern.
The speed is different this time, admittedly. AI capabilities are advancing faster than previous technologies, and the potential scope is broader. But markets adapt faster when the stakes are higher, and the stakes have never been higher. The same forces that drove rapid agricultural innovation in the face of predicted famine will drive even faster adaptation in the face of AI disruption.
I’ve seen this panic up close. My middle child, who has strong math skills and is a thoughtful problem solver, is planning to earn a Master’s in Computer Engineering. He asked if that was a mistake. I told him no. Hot takes at this scale are almost always wrong. The system adapts in ways first-order forecasts miss, and the people who understand the tools are the ones who thrive when it does.
Doom sells better than nuance. “AI will end all jobs” gets more clicks than “jobs will change in unpredictable ways.” Hot takes spread because they’re clean and simple. But complexity is where the truth lives, and where the opportunity hides.
In the 1960s, the refrain was “Don’t have kids, the world will starve.” Today, it’s “Don’t learn to code, AI will do it all.” Both ignore the same truth, when there’s money to be made, markets adapt, and the winners are the ones who adapt with them.
How well-intentioned automation traps people in frustrating loops, and what we can do to stop it.
My wife is from Belarus. On one of my first visits there, I had my first real exposure to what extreme bureaucracy looked like.
Each time I visited a new city, if I stayed more than a certain number of days, I had to register with the police. The process could take an entire day and involved going to a bank to deposit money into the police branch’s account, then returning with a receipt.
One time, we tried to withdraw the remaining cash and close a bank account. We spent the entire day waiting in line after line, at one bank location after another. In the end, we gave up because the opportunity cost was greater than the amount of money we were trying to reclaim.
Need to pay for passport photos? You could not pay in cash, I assume due to fear of fraud and graft; you had to go to the bank, transfer money to the photo shop, and bring back a receipt to prove it.
What struck me was that I was the only one who found this painful. Everyone else accepted it as normal. Endless lines, paperwork, and procedural steps that seemed arbitrary and counterproductive.
So why am I writing about this? This morning I was reflecting on recent experiences changing flights and helping my parents with their Comcast subscription. Over and over, I ran into automation that was supposed to make things easier but actually made things worse.
I tried to change a flight from London to Seattle on Delta. Since it was a codeshare with Virgin, the website couldn’t handle it. I called the support line and got pushed through a phone tree that aggressively tried to send me back to the website. The site still didn’t work. I called back and asked to speak with someone and was routed to a virtual assistant that did nothing but run keyword searches on the help site. Eventually, I got connected to a lower-tier agent who told me the $350 fare difference I saw online wasn’t correct and that it would be $3,000. I pushed back until I reached someone who could actually help. They made the change. The entire process took nearly two hours.
Then there was Comcast. My aging parents have been living with me, and I’ve been helping with their bills. I noticed their TV and internet service had crept up to $350 per month. It was the result of expired deals, supposed discounts that added phone lines they never used, and a long list of tactics designed to get people to pay more for services they didn’t want. Fixing it took well over an hour, and once again I had to fight through automation before talking to someone who would do anything.
Not all automation becomes bureaucracy. My USAA mobile app lets me deposit checks instantly, transfer money in two taps, and reach a human agent with a button press when something goes wrong. It avoids the bureaucracy trap because it was designed around what I actually need to do, with seamless escalation when the automation isn’t enough.
There’s a saying that comes to mind: don’t attribute to malice what can be explained by ignorance. The people who built these systems were probably trying to help. But they were judged by what they shipped and time saved on support calls, not by whether their systems improved user experience.
So what does this mean? When we build systems like these, we need to start by deciding how we will define and measure success. That question should come at the beginning, not the end. It needs to shape how the system is designed, not just how it is reported.
Too often, we optimize for metrics that are easy to measure, like time on call or tickets closed, rather than the experience we’re actually trying to create. Instead, consider measuring success by how empowered customers feel, not how fast they hang up. Once the system is live, we have to come back and test our assumptions. That means checking whether it actually helps users, not just whether it saves time or reduces support volume. One way to do this is to regularly review customer satisfaction and compare it to the experience we intended to create. If it isn’t working, we need to change how the system behaves and what we measure.
This is especially important as we start building with AI. These systems can develop unexpected behaviors. Take Air Canada’s chatbot, which confidently told a customer he could buy a full-price ticket to his grandmother’s funeral and apply for a bereavement discount within 90 days after travel. This was completely wrong. When the customer tried to get the promised refund, the airline refused and even argued the chatbot was a “separate legal entity responsible for its own actions.” Unlike a phone tree that just frustrates you, the AI gave authoritative-sounding but fabricated policy information. The airline probably measured success by how many conversations the AI handled without escalating to humans, not realizing that customers who got wrong answers often just give up rather than keep fighting.
What we choose to measure and how fast we respond when something goes wrong matters more than ever. Once these systems are deployed, they don’t just carry our assumptions forward. They reinforce them. They make it harder to see when the original design was flawed, because the automation itself becomes the norm.
The goal should always be to reduce friction and make life easier for real people, not just to make things more efficient for the teams who built the system. The best systems I’ve used made it easy to talk to a human when I needed to, and didn’t treat automation as a wall to hide behind.
If we lose sight of that, we risk recreating the same kind of bureaucracy I saw years ago, only now it will be faster, more rigid, and much harder to argue with.
I was doing some work on readying a launch for our integration with mDL authentication into one of our products when I realized I finally had to deal with the patchwork of state support. California? Full program, TSA-approved, Apple Wallet integration. Texas? Absolute silence. Washington state, practically ground zero for tech, somehow has nothing.
At a glance the coverage made no sense until I started thinking deeper. Turns out we accidentally ran the largest identity verification stress test in history, and only some states bothered learning from it.
Between 2020-2023, fraudsters systematically looted $100-135 billion from unemployment systems using the most basic identity theft techniques. The attack vectors were embarrassingly simple: bulk-purchased stolen SSNs from dark web markets, automated claim filing, and email variations that fooled state systems into thinking [email protected] and [email protected] were different people.
What nobody anticipated, this massive failure would become the forcing function for digital identity infrastructure. Here’s the thing about government security. Capability doesn’t drive adoption, pain does. The Real ID Act passed in 2005. Twenty years later, we’re still rolling it out. But lose a few billion to Nigerian fraud rings? Suddenly digital identity becomes a legislative priority.
The correlation is stark:
State
Fraud Losses
mDL Status
California
$20-32.6B
Comprehensive program, Apple/Google integration
Washington
$550-650M
Nothing (bill stalled)
Georgia
$30M+ prosecuted
Robust program, launched 2023
Texas
Under $1B estimated
No program
New York
Around $1-2B
Launched 2025
States that got burned built defenses. States that didn’t, didn’t. This isn’t about technical sophistication. Texas has plenty of that. It’s about the political will created by public humiliation. When your state pays unemployment benefits to death row inmates, legacy approaches to remote identity verification stop being defensible.
Washington is the fascinating outlier. Despite losing over $1 billion and serving as the primary target for international fraud rings, they still have no mDL program. The bill passed the Senate but stalled in the House. This tells us something important: crisis exposure alone isn’t sufficient. You need both the pain and the institutional machinery to respond.
The timeline reveals the classic crisis response pattern. Fraud peaked 2020-2022, states scrambled to respond 2023-2024, then adoption momentum stalled by mid-2024 as crisis memory faded. But notice the uptick in early 2025—that’s Apple and Google entering the game.
In December 2024, Google announced its intent to support web-based digital ID verification. Apple followed with Safari integration in early 2025. By June, Apple’s iOS 26 supported digital IDs in nine states with passport integration. This shifts adoption pressure from crisis-driven (security necessity) to market-driven (user expectation).
When ~30% of Americans live in states with mDL programs and Apple/Google start rolling out wallet integration this year, that creates a different kind of political pressure. Apple Pay wasn’t crisis-driven, but became ubiquitous because users expected it to work everywhere. Digital identity in wallets will create similar pressure. States could rationalize ignoring mDL when it was ‘just’ about fraud prevention. Harder to ignore when constituents start asking why they can’t verify their identity online like people in neighboring states.
We’re about to find out whether market forces can substitute for crisis pressure in driving government innovation. Two scenarios. Consumer expectations create sustainable political pressure, and laggard states respond to constituent demands. Or only crisis-motivated states benefit from Apple/Google integration, creating permanent digital divides.
From a risk perspective, this patchwork creates interesting attack surfaces. Identity verification systems are only as strong as their weakest links. If attackers can forum-shop between states with different verification standards, the whole federation is vulnerable. The unemployment fraud taught us that systems fail catastrophically when overwhelmed.
Digital identity systems face similar scalability challenges. They work great under normal load, but can fail spectacularly during a crisis. The states building mDL infrastructure now are essentially hardening their systems against the next attack.
If you’re building anything that depends on identity verification, this matters. The current patchwork won’t last; it’s either going to consolidate around comprehensive coverage or fragment into permanent digital divides. For near-term planning, assume market pressure wins. Apple and Google’s wallet integration creates too much user expectation for politicians to ignore long-term. But build for the current reality of inconsistent state coverage.
For longer-term architecture, the states with robust mDL programs are effectively beta-testing the future of government digital services. Watch how they handle edge cases, privacy concerns, and technical integration challenges.
We accidentally stress-tested American federalism through the largest fraud in history. Only some states learned from the experience. Now we’re running a second experiment: can consumer expectations accomplish what security crises couldn’t?
There’s also a third possibility. These programs could just fail. Low adoption rates, technical problems, privacy backlash, or simple bureaucratic incompetence could kill the whole thing. Government tech projects have a stellar track record of ambitious launches followed by quiet abandonment.
Back to my mDL integration project: I’m designing for the consumer pressure scenario, but building for the current reality. Whether this becomes standardized infrastructure or just another failed government tech initiative, we still need identity verification that works today.
The criminals who looted unemployment systems probably never intended to bootstrap America’s digital identity infrastructure. Whether they actually succeeded remains to be seen.
I’ve been building with computer vision and ML since before it was cool, and I use these tools daily. When my middle child announced they were majoring in computer engineering, I didn’t panic about automation taking their job. I encouraged it.
But something strange is happening in the world of software development that very few seem to be talking about. AI has created a paradox: building software has never been more accessible, yet the risks of ownership have never been higher.
This pattern isn’t isolated to software development. I’ve been tracking similar dynamics across compliance, skill markets, and organizational structures. In each domain, AI is creating the same fundamental shift: execution becomes liquid while orchestration becomes critical. The specific risks change, but the underlying forces remain consistent.
When Building Feels Free
Aaron Levie from Box recently laid out the case for where AI-generated software makes sense. His argument is nuanced and grounded in Geoffrey Moore’s framework of “core” versus “context” activities. Most people don’t actually want to build custom software, he argues, because they’re fine with what already works and “the effort to customize is greater than the ROI they’d experience.” Taking responsibility for managing what you build is “often not worth it” since when something breaks, “you’re on your own to figure out what happened and fix it.”
More fundamentally, custom development is simply less efficient for most business problems. Context activities like payroll, IT tickets, and basic workflows are things “you have to do just to run your organization, but a customer never really notices.” You want to spend minimal time maintaining these systems because no matter how well you execute them, customers rarely see the difference.
The real opportunity, Levie argues, lies elsewhere – integration work between existing systems, custom optimizations on top of standard platforms for organizational edge cases, and addressing the long tail of core business needs that have never been properly served. The result will be “an order of magnitude more software in the world,” but only where the ROI justifies customization.
Levie’s right, but he’s missing something crucial. AI isn’t just making certain types of custom development more viable. It’s fundamentally changing what ownership means. The same technology that makes building feel effortless is simultaneously making you more liable for what your systems do, while making evaluation nearly impossible for non-experts.
When Your Code Becomes Your Contract
Air Canada learned this the hard way when their chatbot promised a bereavement discount that didn’t exist in their actual policy. When their customer tried to claim it, Air Canada argued the chatbot was a “separate legal entity” responsible for its own actions.
The Canadian tribunal’s response was swift and unforgiving. They called Air Canada’s defense “remarkable” and ordered them to pay. The message was clear: you own what you deploy, regardless of how it was created.
This isn’t just a one-off, global regulations are tightening the screws on software accountability across every jurisdiction. The EU’s NIS2 directive creates real liability for cybersecurity incidents, with fines up to $10.8 million (€10 million) or 2% of global turnover. SEC rules now require public companies to disclose material incidents within four business days. GDPR has already demonstrated how quickly software liability can scale. Meta faced a $1.3 billion (€1.2 billion) fine, and Amazon got hit with $806 million (€746 million).
While these are not all AI examples, one thing is clear. When your AI system makes a promise, you’re bound by it. When it makes a mistake that costs someone money, that’s your liability. The technical complexity of building software has decreased, but the legal complexity of owning it has exploded into a ticking time bomb.
AI’s Hidden Danger
Here’s where the paradox gets dangerous. The same AI that makes building feel effortless makes evaluation nearly impossible for non-experts. How do you test a marketing analytics system if you don’t understand attribution modeling? How do you validate an HR screening tool if you can’t recognize algorithmic bias?
This creates what I call “automation asymmetry” – the same dynamic I’ve observed in compliance and audit workflows. AI empowers the builders to create sophisticated, polished systems while leaving evaluators struggling with manual review processes. The gap between what AI can help you build and what human judgment can effectively assess is widening rapidly.
As a product leader, I constantly weigh whether we can create enough value to justify engineering, opportunity costs, and maintenance costs. AI makes this calculation treacherous. A feature that takes a day to build with AI might create months of hidden maintenance burden that I can’t predict upfront. The speed of development is now disconnected from the cost of ownership.
Unlike traditional software bugs that fail obviously, AI systems can exhibit “specification gaming.” They appear to work perfectly while learning fundamentally wrong patterns.
This is Goodhart’s Law in action. When a measure becomes a target, it ceases to be a good measure. The AI system learns to optimize for your evaluation criteria rather than the real-world performance you actually care about.
Picture an HR screening system that correctly identifies qualified candidates in testing but starts filtering out good applicants based on subtle biases in the training data in the foundation model you built on. This isn’t a bug you can catch with normal testing. It requires understanding algorithmic bias that most organizations lack.
Or consider data leakage risks. AI systems can inadvertently memorize and leak personal information from their training data, but detecting this requires privacy testing that most organizations never think to perform. By the time you discover your customer service bot is occasionally revealing other users’ details, you’re facing GDPR violations and broken customer trust.
Imagine a bank that “validates” its loan applications analysis by testing on the same templates used for training. They celebrate when it passes these tests, not understanding that this proves nothing about real-world performance. Or consider a logistics company that builds an AI route optimization system. It works perfectly in testing, reducing fuel costs by 15%. But after deployment, it makes decisions that look efficient on paper while ignoring practical realities. It routes through construction zones, sends drivers to nonexistent addresses, and optimizes for distance while ignoring peak traffic patterns.
Many ownership challenges plague all custom software development. Technical debt, security risks, staff turnover, and integration brittleness. But AI makes evaluating these risks much harder while making development feel deceptively simple. Traditional software obviously fails. AI software can fail silently and catastrophically.
Why Even Unlimited Resources Fail
Want proof that ownership is the real challenge? Look at government websites. These organizations have essentially unlimited budgets, can hire the best contractors, and have national security imperatives. They still can’t keep basic digital infrastructure running.
The Social Security Administration’s technical support runs Monday through Friday, 7:30 AM to 4:00 PM Eastern. For a website. In 2025. Login.gov schedules multi-hour maintenance windows for essential services. Georgetown Law Library tracked government URLs from 2007 and watched half of them die by 2013. Healthcare.gov cost $2.1 billion and barely worked at launch.
These aren’t technical failures. They’re ownership failures. Many government projects falter because they’re handed off to contractors, leaving no one truly accountable yet the agencies remain liable for what gets deployed. The same organizations that can build nuclear weapons and land rovers on Mars can’t keep websites running reliably, precisely because ownership responsibilities can’t be outsourced even when development is.
“But wait,” you might think, “commercial software companies fail too. What about when vendors go bankrupt and leave customers stranded?”
This objection actually proves the point. When Theranos collapsed, their enterprise customers lost the service but weren’t held liable for fraudulent blood tests. When a SaaS company fails, customers face transition costs and data loss, but they don’t inherit responsibility for what the software did during operation.
Compare that to custom system failure. When your AI medical analysis tool makes a misdiagnosis, you don’t just lose the service. You’re liable for the harm it caused. The failure modes are fundamentally different.
If even professional software companies with dedicated teams and specialized expertise sometimes fail catastrophically, what makes a non-software organization think they can manage those same risks more effectively? If unlimited resources can’t solve the ownership problem, what makes us think AI-generated code will?
Traditional ownership costs haven’t disappeared. They’ve become economically untenable for most organizations. Technical debt still compounds. People still leave, taking institutional knowledge with them. Security vulnerabilities still emerge. Integration points still break when external services change their APIs.
AI makes this trap seductive because initial development feels almost free. But you haven’t eliminated ownership costs. You’ve deferred them while adding unpredictable behavior to manage.
Consider a typical scenario: imagine a marketing agency that builds a custom client reporting system using AI to generate insights from campaign data. It works flawlessly for months until an API change breaks everything. With the original developer gone, they spend weeks and thousands of dollars getting a contractor to understand the AI-generated code well enough to fix it.
These businesses thought they were buying software. They were actually signing up to become software companies.
The New Decision Framework
This transformation demands a fundamental shift in how we think about build versus buy decisions. The core question is no longer about execution capability; it’s about orchestration capacity. Can you design, evaluate, and govern these systems responsibly over the long term?
You should build custom software when the capability creates genuine competitive differentiation, when you have the institutional expertise to properly evaluate and maintain the system, when long-term ownership costs are justified by strategic value, and when existing solutions genuinely don’t address your specific needs.
You should buy commercial software when the functionality is context work that customers don’t notice, when you lack domain expertise to properly validate the system’s outputs, when ownership responsibilities exceed what you can realistically handle, or when proven solutions already exist with institutional backing.
Commercial software providers aren’t just offering risk transfer. They’re developing structural advantages that individual companies can’t match. Salesforce can justify employing full-time specialists in GDPR, SOX, HIPAA, and emerging AI regulations because those costs spread across 150,000+ customers. A 50-person consulting firm faces the same regulatory requirements but can’t justify even a part-time compliance role.
This reflects Conway’s Law in reverse: instead of organizations shipping their org chart, the most successful software companies are designing their org charts around the complexities of responsible software ownership.
Mastering the Paradox
The AI revolution isn’t killing software development companies it’s fundamentally changing what ownership means and repricing the entire market. Building has become easier, but being responsible for what you build has become exponentially harder.
This follows the same pattern I’ve tracked across domains: AI creates automation asymmetry, where execution capabilities become liquid while orchestration and evaluation remain stubbornly complex. Whether in compliance audits, skill markets, or software ownership, the organizations that thrive are those that recognize this shift and invest in orchestration capacity rather than just execution capability.
Advanced AI development tools will eventually solve some of these challenges with better validation frameworks and automated maintenance capabilities. We’ll likely see agentic AI automating much of our monotonous security, support, and maintenance work in the future also. These systems could help organizations build the connective tissue they lack automated monitoring, intelligent debugging, self-updating documentation, and predictive maintenance. But we’re not there yet, and even future tools will require expertise to use effectively.
This doesn’t mean you should never build custom software. It means you need to think differently about what you’re signing up for. Every line of AI-generated code comes with a lifetime warranty that you have to honor.
The question isn’t whether AI can help you build something faster and cheaper. It’s whether you can afford to own it responsibly in a world where software liability is real, evaluation is harder, and the consequences of getting it wrong are higher than ever.
Understanding this paradox is crucial for anyone making build-versus-buy decisions in the AI era. The tools are more powerful than ever, but mastering this new reality means embracing orchestration over execution. Those who recognize this shift and build the institutional capacity to govern AI systems responsibly will define the next wave of competitive advantage.
You’re not just building software. You’re signing up for a lifetime of accountability.
Early in my career, I was a security engineer, in this role we approached problems with a distinctive mindset. We look at a new system and immediately start threat modeling. What could go wrong? Where are the vulnerabilities? How might an attacker exploit this? Then we’d systematically build defenses, design monitoring systems, and create incident response procedures.
Later at Microsoft, I realized that good lawyers operate almost identically. They’re security engineers for text. When reviewing a contract, they’re threat modeling potential disputes. When structuring a transaction, they’re identifying legal vulnerabilities and designing defenses. When arguing a case, they’re building systems to withstand attack from opposing counsel. Of course, not all legal work requires this depth of analysis but the most valuable does.
This realization first drew me to Washington State’s legal apprenticeship program. The idea of learning legal “security engineering” through hands-on mentorship rather than accumulating law school debt appealed to me. I never pursued it, but I remained fascinated by sophisticated legal reasoning, regularly listening to Advisory Opinions and other legal podcasts where excellent legal minds dissect complex problems.
Just as I’ve written about skill liquidity transforming compliance and software development, the same forces are reshaping legal practice. AI is injecting liquidity into what was once an extremely illiquid skill market, where expertise was scarce, slowly accumulated, and tightly guarded. But here’s what’s different, current legal AI treats lawyers like document processors when they’re actually security engineers, and this fundamental misunderstanding is creating unexpected problems.
The Security Engineer’s Dilemma
Legal skills have historically been among the most illiquid in professional markets, exactly the kind of expertise scarcity that AI disruption targets. A single fabricated case citation can end careers. Imagine a security engineer whose monitoring system had a 99% accuracy rate, but the 1% false negatives included critical breaches that were mishandled so badly that the company’s reputation was irreparably damaged. Reputation operates on decades-long timescales because clients hire lawyers based on their track record of successfully defending against legal threats.
Just as I described in software development, AI injecting liquidity into skill markets creates a repricing rather than elimination. Where legal expertise was once scarce and slowly accumulated, AI makes certain capabilities more accessible while reshaping where the real value lies.
Traditional legal training worked like security engineer mentorship. Junior lawyers learned threat modeling by working on real cases under senior guidance. They’d review contracts while learning to spot potential vulnerabilities, draft briefs while understanding how opposing counsel might attack their arguments, structure deals while considering regulatory risks. Quality control and knowledge transfer happened simultaneously, seniors reviewing junior work would catch errors while teaching systematic risk assessment.
AI is disrupting this model in ways that would terrify any security team lead. Document review, research, and drafting that once provided junior lawyers with hands-on threat modeling experience are being automated. The tasks that taught pattern recognition, learning to spot the subtle contract clause that creates liability exposure, recognizing the factual detail that undermines a legal argument, are disappearing.
This creates the same middle tier squeeze I explored in software development, acute pressure between increasingly capable juniors and hyper-productive seniors. Junior lawyers become more capable with AI assistance while partners extend their span of control through AI tools, leaving mid-level associates caught in a compressed middle where their traditional role as the “throughput engine” gets automated away.
Here’s the economic problem, when AI saves 20 hours on document review, partners face a choice between investing those hours in unpaid training or billing them elsewhere. The math strongly favors billing. Fixed-fee arrangements make this worse, junior lawyers become cost centers rather than revenue generators during their learning phase.
The Governance Crisis
Current legal AI focuses on document creation, research assistance, and drafting support, essentially better word processors and search engines. While impressive, they’ve created a governance burden that’s consuming the time that should be spent teaching threat modeling skills.
This mirrors what I wrote about in compliance “automation asymmetry” where sophisticated AI-generated documents overwhelm human review capacity. Just as automated compliance artifacts can mask underlying issues with perfect formatting, AI legal documents create a veneer of competence that requires more, not less, senior oversight.
Every AI-generated document requires human verification. Partners must create AI policies, review procedures, and verification systems. The American Bar Association found lawyer concerns about AI accuracy increased from 58% in 2023 to 75% in 2025, exposure to these tools has heightened rather than reduced professional anxiety. Firms now maintain an average of 18 different AI solutions, each requiring its own governance framework.
This is like asking security engineers to spend their time verifying automated log reports instead of designing security architectures. Quality control has shifted from collaborative threat assessment to bureaucratic oversight. Instead of senior lawyers working through junior analysis while explaining legal reasoning, we have senior lawyers checking AI output for fabricated cases and subtle errors.
The teaching moments are gone. The efficient combination of quality control and knowledge transfer that characterized traditional review has been broken into separate activities. Senior expertise gets consumed by managing AI rather than developing human threat modeling capabilities.
There’s a deeper concern too. Security engineers know that over-reliance on automated tools can weaken situational awareness and pattern recognition. Legal reasoning requires the same kind of layered understanding, knowing not just what the law says, but how different doctrines interact, how factual variations affect outcomes, how strategic considerations shape arguments. AI can provide correct answers without fostering the threat modeling instincts that distinguish excellent lawyers.
The problem isn’t that we have AI in legal practice, it’s that we have the wrong kind of AI.
Building Better Security Tools
The fundamental problem is architectural. Current legal AI treats legal work as document processing when it’s actually systematic threat assessment. Most legal AI focuses on output, drafting contracts, researching case law, generating briefs. This misses the intellectual core, the systematic risk analysis frameworks that constitute legal reasoning.
Good lawyers, like security engineers, rely on systematic approaches. Constitutional analysis follows specific threat models for government overreach. Contract law has systematic frameworks for identifying formation vulnerabilities, performance risks, and breach scenarios. Tort analysis uses systematic negligence assessment patterns. These frameworks require internalization through guided practice that current AI disrupts.
But imagine different AI, tools designed for threat modeling rather than document creation. Instead of generating contract language, AI that helps identify potential vulnerabilities in proposed terms. Instead of researching cases, AI that systematically maps the legal threat landscape for a particular situation. Instead of drafting briefs, AI that helps build comprehensive defensive arguments while teaching the reasoning patterns that make them effective.
This would change governance entirely. Instead of verifying AI-generated content, lawyers would verify AI-enhanced threat assessments. Systems that show their analytical work (explaining why certain contract clauses create liability exposure, how different factual scenarios affect legal outcomes) enable both quality control and learning.
Security engineers don’t just need better log parsing tools; they need better threat modeling frameworks. Lawyers face the same challenge. The 19th-century apprenticeship model worked because it focused on developing systematic risk assessment capabilities through guided practice with real threats.
The Pattern Continues
This completes a progression I’ve traced across professions, and the pattern is remarkably consistent. In software development, execution capabilities are being liquified, but systems architects who understand complex threat models gain value. In compliance, process expertise is being liquified, but systematic thinkers who can model regulatory interactions across domains have advantages.
In law, legal reasoning itself could be liquified, but the outcome depends on whether we develop AI that enhances threat modeling capabilities rather than just automating document production. The sophisticated legal analysis I love hearing on Advisory Opinions represents systematic risk assessment applied to complex problems. This is exactly the kind of security engineering thinking that creates real value.
The pattern across all three domains is clear, as AI makes execution more liquid, value shifts toward orchestration. In software, orchestrators who build AI-augmented workflows and internal platforms create structural advantages. In compliance, orchestrators who design intelligent systems for continuous assurance gain leverage over reactive, manual approaches.
Current legal AI accidentally creates governance overhead that eliminates mentorship. But reasoning-focused AI could enhance both efficiency and competence development. Instead of making lawyers better document processors, we could make them better security engineers, orchestrators who design systematic threat assessment capabilities rather than just executors who handle individual risks.
The choice matters because society depends on lawyers who can systematically identify legal threats and build robust defenses. Current AI accidentally undermines this by turning lawyers into document reviewers instead of security architects.
The Washington apprenticeship program that first caught my attention represents something important, learning through guided practice with real threats rather than theoretical study. The future may not eliminate apprenticeship but transform it, the first generation learning legal threat modeling through AI designed to build rather than replace systematic reasoning capabilities.
When I listen to Advisory Opinions, I’m hearing security engineers for text working through complex threat assessments. That’s the kind of thinking we need more of, not less. We can build AI that enhances rather than replaces it.
I advised Let’s Encrypt from its early days, watching it transform the security foundation of the web. Most think it won by offering free certificates. That’s dead wrong.
Existing CAs had already enabled free certificates years earlier. GlobalSign’s CloudSSL API, launched in 2011, (in full disclosure, I was their CTO), provided the automation that allowed Cloudflare to offer free SSL to end users; other CAs offered free short-lived certificates as part of forever trials as well. By 2015, you could buy DV certificates for $3-5 from certificate resellers, it was clear people were willing to pay for support which is largely what these resellers offered. The real story is about organizational constraints and misaligned incentives.
Conway’s Law Explains Everything
Traditional certificate authorities were trapped by their own organizational structure. Their business model incentivized vendor lock-in rather than ecosystem expansion and optimization. Sales teams wanted products’ proprietary APIs to make it harder for customers to switch, and were riding the wave of internet expansion. Compliance teams’ jobs depended on defending existing processes. Engineering teams were comfortable punting all compliance work to the “compliance” department. Support teams were positioned as competitive differentiators and used to entrench customers. Their goal was maximizing revenue, defending their jobs, and maintaining the status quo, not getting the web to 100% HTTPS.
Let’s Encrypt had completely different incentives and could optimize solving the larger problems without these organizational constraints. But LE’s success went beyond solving their own problems. They systematically identified every pain point in the way of getting to 100% HTTPS and built solutions that worked for everyone.
What LE Could Do That Traditional CAs Couldn’t
True standardization. Before ACME (the protocol that automates certificate requests), every major CA had incompatible automation systems. Comodo, DigiCert, GlobalSign and others each had proprietary approaches that required custom integration and as a result, had inherent switching costs; they saw no incentive to work together to standardize as a result. LE led the creation of ACME as an open standard that made switching CAs as simple as changing a configuration setting.
This enabled applications like Caddy and Google Cloud Load Balancer to handle certificates automatically for their customers without vendor-specific code. Once cloud platforms could flip switches to HTTPS-by-default, network effects became unstoppable.
Ecosystem-wide solutions. When LE felt coordination pain from renewal spikes and incident-related revocations, they created ACME Renewal Information (ARI, a protocol extension that helps coordinate renewal timing) so all CAs could prevent renewal storms. Traditional CAs couldn’t build these solutions because their org charts prevented optimizing for competitors’ success and instead focused on riding the internet expansion.
Engineering-driven compliance. Instead of compliance teams reviewing certificates after issuance, LE built policy compliance directly into certificate generation pipelines. Violations became orders of magnitude harder rather than detectable. Traditional CAs couldn’t eliminate their compliance departments because those jobs justified organizational overhead.
The Market Found Natural Segments
Mozilla telemetry reveals exactly what happened. Let’s Encrypt dominates issuance at 46.1% of certificates but ranks third in Firefox usage. LE democratized HTTPS for the long tail: domain parking networks, no-code builders, shared hosting platforms serving millions of low-traffic sites.
Meanwhile, high-traffic sites gravitated toward CAs like Google Trust Services (in full disclosure, I was responsible for creation of this service) that lead usage, as its used by large sites that value high availability and performance, leading to more relying party reliance despite lower issuance volumes, or established players like DigiCert and Sectigo that focus on supporting large enterprise customers. These sites need commercial support and accountability when things go wrong. The market is segmented around operational needs: the long tail valued automation over accountability, while major platforms needed enterprise support and someone to support them when something goes wrong.
Once long-tail providers flipped to HTTPS-by-default, encrypted pages became the norm. Google’s Transparency Report shows 99% of Chrome page-loads now occur over HTTPS, a transformation that began when Let’s Encrypt launched in April 2016.
The Industry Finally Admitted LE Was Right
Here’s the ultimate vindication: in 2025, the CA/Browser Forum mandated 47-day maximum certificate validity by 2029, with Chrome requiring automation from every public CA. Let’s Encrypt didn’t follow industry trends. The industry now follows Let’s Encrypt.
What seemed like LE’s “unusual” 90-day lifespans in 2016 became conservative by 2025. The mandate’s technical reasoning mirrors what LE pioneered: short-lived certificates reduce dependence on revocation checking, reduce key compromise windows, and force automated resilient infrastructure.
Leading organizations moved even further ahead. Netflix runs 30-day certificates in production, Google issues 7-day certificates for infrastructure, and Let’s Encrypt will introduce 6-day certificates by end of 2025. The mandates aren’t pushing innovation forward; they’re codifying where leaders already operate.
Why This Matters Beyond Certificates
Let’s Encrypt proved that critical internet infrastructure could be reimagined from first principles rather than optimized around legacy organizational constraints and practices. But the implications go deeper than certificate automation.
Traditional CAs were fundamentally vetting authorities with deep expertise in legal requirements for vetting people and businesses worldwide. They should have owned the remote identity verification market that exploded with digital transformation. Instead, they remained myopically focused on public trust-based certificate products while companies like Jumio and Onfido captured those opportunities. At the same time, they missed the massive expansion of machine and workload identity because they were ignoring private PKI use cases. They weren’t just leaving money on the table; they were failing to build a resilient business and neglecting the foundation for the trust infrastructure they supposedly managed.
The same organizational constraints that prevented CAs from building ACME also blinded them to adjacent markets that were natural extensions of their core competencies. They were too focused on maintaining certificate revenue streams and too constrained by existing structures to recognize how the world was shifting from hosting providers to cloud to SaaS.
ACME became the standard not because it was technically superior to existing APIs, though it was, but because it was designed for portability rather than lock-in. ARI emerged because LE experienced ecosystem pain and could fix it without navigating corporate bureaucracy or competitive concerns.
The complexity and friction we’d accepted for decades weren’t inherent to certificate management. It was the byproduct of organizational structures optimizing for vendor revenue rather than user adoption.
Today’s 47-day mandate represents more than policy evolution. It’s the industry formally acknowledging that Let’s Encrypt defined the correct approach for internet trust infrastructure. Conway’s Law isn’t destiny, but escaping it requires the courage to rebuild systems around user needs rather than organizational convenience.
